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To meet customer expectations and remain competitive, industrials try 
constantly to improve their quality control systems. There is hence 
increasing demand for adopting automatic defect detection solutions. 
However, the biggest issue in addressing such systems is the imbalanced 
aspect of industrial datasets. Often, defect-free samples far exceed the 
defected ones, due to continuous improvement approaches adopted by 
manufacturing companies. In this sense, we propose an automatic defect 
detection system based on one-class classification (OCC) since it involves 
only normal samples during training. It consists of three sub-models, first, a 
convolutional autoencoder serves as latent features extractor, the extracted 
features vectors are subsequently fed into the dimensionality reduction 
process by performing principal component analysis (PCA), then the 
reduced-dimensional data are used to train the one-class classifier support 
vector data description (SVDD). During the test phase, both normal and 
defected images are used. The first two stages of the trained model generate 
a low-dimensional features vector, whereas the SVDD classifies the new 
input, whether it is defect-free or defected. This approach is evaluated on the 
carpet images from the industrial inspection dataset MVTec anomaly 
detection (MVTec AD). During training, only normal images were used. The 
results showed that the proposed method outperforms the state-of-the-art 
methods. 
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1. INTRODUCTION 


Efficient quality inspection is one of the cornerstones of successful manufacturing companies. Since 
human visual inspection is error-prone and subjective, there has been a movement towards automatic defect 
detection systems [1], more than that, recently we witness the transition to the era of quality 4.0, which we 
can define as a mature quality system that sought to leverage industry 4.0 technologies. In fact artificial 
intelligence and machine learning [2] had proven their abilities to perform quality inspection [3]. A plethora 
of defect detection methods have shown to be promising and efficient [4]. However most of them are based 
on supervised learning [5], requiring considerable amount of normal and abnormal (i.e. defected) samples to 
train the model efficiently, which is not always available in real applications. Actually, manufacturing 
companies adopt increasingly improvement approaches [6] as lean management and six sigma [7], with the 
attention of managing the fabrication process and reducing defective products by targeting zero defect 
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manufacturing. Defects hence rarely occur in production, and meanwhile there is enormous need to build 
automatic defect detection system. 

One-class classification (OCC) algorithms [8] offer the potential to create automatic inspection 
system in early stage without having to wait for more defected samples to be collected. Consequently 
industrials could exploit the important amount of data in their possession, despite highly skewed class 
distribution. The main idea of OCC methods is to distinguish between normal and abnormal class, drew on 
the knowledge gained of normal samples during training. 

The term OCC was first introduced by [9] to denote a category of classification algorithms that 
address cases where few to none defect samples are available for training; the normal class is well-defined 
while abnormal one is under-sampled [10] which is quite common in industrial areas [11] ,and with that, 
defects are seen as a deviation from defect-free class. The OCC concept encompasses several approaches, 
such as methods based on density [12], distance [13], neural networks [14], [15], and boundary approaches 
[16] that aims to encircle normal samples by a decision boundary. The work [17] developed adversarially 
trained deep neural networks, the first component works as image reconstructor, while the second represents 
the classifier. Autoencoders also were used to address OCC problems [18], [19]. In general, these researchers 
assumed that autoencoders generates higher reconstruction error for defected samples. Going through 
research works that address OCC, it is clearly apparent that support vector data description (SVDD) [20] is 
one of the extensively used algorithms in OCC applications for its satisfactory results [21]. Luo et al. [22] 
introduced a cost-sensitive SVDD, which is a method that provides different costs to classification errors. 
The paper [23] developed a new SVDD approach to deal with uncertain data. They trained the SVDD by 
using the similarity scores between examples and normal class. It is reported to outperforms regular SVDD in 
terms of sensitivity to noise. Shi et al. [24] introduced improved SVDD by combining relative density weight 
with SVDD. However, those SVDD based methods have difficulty dealing with large and high-dimensional 
datasets [25] mainly because of optimization complexity. 


2. METHOD 

The intent of this work is to create an automatic defect detection system. We propose a model that 
could manage imbalanced data and, at the same time yield to interesting results in high-dimensional spaces 
using SVDD algorithm. In this section, we describe the proposed defect detection system in detail, and we 
highlight the three components of our model: convolutional autoencoder, principal component analysis 
algorithm (PCA) and SVDD algorithm. 


2.1. Overview 

Product surface images are used as input for our algorithm and only images of normal class are 
present during training. The proposed system consists of three main phases. Firstly, we use a convolutional 
autoencoder that allows extracting image’s abstract features. Once this submodel is trained, the decoder part 
is discarded. Then the bottleneck features vector is later fed into PCA in order to perform dimensionality 
reduction by inducing efficient and discriminating features representation. So that it serves as training input 
of the SVDD classifier. The test images are forwarding through the trained model to determine whether the 
image is normal or defected. The proposed system takes advantage of the convolutional autoencoder ability 
to extract robust features automatically; meanwhile it alleviates the problem of SVDD of handling high- 
dimensional datasets. Figure 1 shows the overview of the proposed approach, and Figure 2 illustrates the 
flowchart of the entire system. 
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Figure 1. Shows the overview of the suggested approach 
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Figure 2. The flowchart of the proposed system 


2.2. Features extraction 

As a first stage a convolutional autoencoder (CAE) is used to extract image features. CAE can 
automatically provide powerful feature learning [26]. Moreover, in [27] it has been proven that the more the 
architecture of convolutional autoencoder becomes deeper, the more powerful features are extracted through 
layers , in a way that first layer provides generic features while subsequent layers extract complex features. 
Basically, autoencoder is a neural network that consists of two connected sub-models: an encoder and a 
decoder. The symmetric structure of the autoencoder allows the reconstruction of input data through the 
features vector provided by the bottleneck layer (i.e., encoder output) as illustrated in Figure 1. Considering 
X the domain of the input data samples and Z the domain of the encodings. 

Given a sample x € X, z € Z the encoded representation of x, and x, the reconstructed input. The 
encoder-decoder process can be summarized, 


Z = ge(x; Oe) (1) 
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X, = Ga(z; Od) (2) 


Where g,: X > Zis the encoding function and g,: Z — X the decoding function, while 0e, Od represent the 
encoder and decoder parameters respectively. A loss function is used to measure reconstruction error; we 
have chosen 2 loss for its simplicity and computational speed. The loss function is formulated as, 


L(x, x;) = I|x a ll (3) 


In this work, we trained the convolutional autoencoder on defect-free samples. Once the model is 
trained, we use the bottleneck layer as automatic features extractor. The encoder consists of five convolution 
layers where each layer is followed by a batch normalization layer. The max-pooling layer is used from the 
second convolution layer where each layers use the rectifier linear unit (ReLu) acti-vation function, 

i. The first two convolutional layers consist of 32 fil-ters of size 3x3, the second one followed by a 
downsampling (max-pooling) layer. 

ii. The third convolutional layer consists of 64 filters of size 3x3, followed by another downsampling 
layer. 

iii. The fourth convolutional layer consists of 128 fil-ters of size 3x3. 

iv. The fifth convolutional layer consists of 256 filters of size 3x3. 


2.3. Dimensionality reduction 

Since SVDD has difficulty handling high-dimensional datasets, PCA [28] was applied to reduce the 
dimensionality of features vectors provided by the CAE. PCA is a statistical procedure that aims to project 
high-dimensional input data into a lower dimension space while retaining most of the information. Let Z be 
an nxm data matrix, where the rows represent the n extracted vectors while the features are represented by 
columns. PCA process is formulates, 
a) Standardize the m-dimensional data 


Zig-—U 


Zig = x (4) 


Here: 2;; is the (¢ )th entry of Z,i =1,..,nand7 =1,...,m. wand s; are respectively the mean 


and the variance of #°™° dimension. 
b) Calculate the covariance matrix Z, 


Five grZ (5) 
n 

c) Construct the n eigenvalues and n eigenvectors via covariance matrix Z, 

ZeVi = AiVi (6) 

Where A; denotes the eigenvalues, v; represents the eigenvectors andi = 1,...,n. 

a. Choose k eigenvectors corresponding to the k highest eigenvalues k<m. 
b. — Project the input data into new k—dimensional space. 

V = [V4, V2) + Vi] (7) 
Then, 

Zz, =V'z (8) 
Where z,’ is the low-dimensional features vector. 
2.4. Defect detection 

The defect detection task is done using SVDD. This algorithm attempts to determine a hypersphere 
with the minimum volume that encircles almost all training data. The spherical boundary is characterized by 


a center a and a radius R, hence during test, points that fall outside the boundary are considered as abnormal 
as illustrated in Figure 3. The parameters R and a are defined by (9), 
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min R*? +C 1, &; (9) 
Subject to, 

lz,’ —all?< R?+%,vi (10) 

& 2ovi (11) 


here €; : are slack variables that allow some points in training data to be outside the sphere and C represents a 
penalty constant that controls the trade-off between the volume of the hypersphere and rejected points. 
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Figure 3. Illustration of SVDD 


The optimization problem (9) is solved via its dual formulation, 


max Yiji1 0}(Zj,2;) — Lig Mi 04(Zj, 24) (12) 
S.t. 

0<a, <Ci=l,2,....n (13) 

Yi-1% = 1 (14) 


Where a; are lagrange multipliers, and (z;,z;) denotes the inner product of z;, and zj.The hypersphere 
boundary is determined by support vectors for which 0 < a; < C. The center ‘a’ is calculated as, 


a= 742.0, (15) 
The radius R is then computed by selecting an arbitrary support vector Z,, 

R* = ||z, — all? = (2p,2,) — 2D a2 21) + Dig Cia (Zi, Z;) (16) 
For each new input y, the distance between this tested point and the center a is computed as, 

dist?(y) = (y,y) — 20% ay ,2;) + Dig 0404(Z;, 24) (17) 


Thus, points with dist?(y) = R? are considered as defected. 
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According to [20], replacing the inner product (zj,z;) in (12) with appropriate kernel function 
K (zi, 2} provides more flexibility to define the boundary. We have chosen to use the Gaussian kernel 


function, given that it is the kernel reported to be yielding satisfactory results in OCC applications [29]. The 
Gaussian kernel is defined as, 


= ror? 
Zi-Zj 


202 


K (z,,2;) = exp (18) 


Where a is the kernel width. 


3. RESULTS AND DISCUSSION 

The experimental environment is a computer with an 17-7700HQ CPU, 16GB of RAM, Nvidia RTX 
2080 Ti GPU, running Windows 10, and we used the Tensorflow library to implement the proposed 
approach. To demonstrate the effectiveness of our suggested system for defect detection, we conducted a set 
of experiments on the carpet images from MVTec anomaly detection (MVTec AD) dataset [30],which is a 
benchmark of natural images dedicated to unsupervised anomaly detection that mimics real industrial 
inspection applications. The carpet dataset consists of 390 high-resolution images of 1024x1024 pixels, 
divided into 72% of normal data and 28% of abnormal data .The training set is composed of defect-free 
images, while the test set consists of normal images as well as images containing 5 different types of fine- 
grained anomalies on the carpet’s surface like threads, cuts, metal contamination, color and holes. The 
dataset overview is shown in Figure 4. Data augmentation was needed to diversify the training data set and 
make the model more generalizable. To do so, random flips and rotations strategies were applied generating 
1449 image. Finally, both categories were rescaled to 512x512 pixels. 


Figure 4. Images of carpet dataset 


We adopted the accuracy, area under the receiver operating characteristic curve (AUROC) and F1 
score to measure the performance of the proposed method. Accuracy and F1 score are computed as, 


Precision = —— (19) 
TP+FP 
Recall = —"— (20) 
TP+FN 
TP+TN 
Accuracy ST PETN REP LEN. (21) 
Pisce =. Precision.Recall (22) 


Precision+Recall 


Where true positive (TP): is the number of images correctly classified as defective. 
True negative (TN): is the number of images correctly classified as normal. 
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False positive (FP): is the number of images incorrectly classified as defective. 
False negative (FN): is the number of images incorrectly classified as normal. 

For this implementation, we assumed that there are no outliers in training data. We defined C=1, 
which implied that € (i=)=0 [31]. The kernel width o is optimized through cross validation. We investigated 
the performance of our model on the carpet images and compare it with state-of-the-art methods. 
Furthermore, we conducted comparison with the two-stage model “Convolutional autoencoder-SVDD” 
(CAE-SVDD), to prove the importance of PCA algorithm in the proposed system. 

As shown in Table 1, the proposed approach method outperforms the compared methods on all 
metrics at detecting the defects in carpet dataset. The AUROC result of CAE-PCA-SVDD model was better, 
with mean value 0.20% higher than the other algorithms. As can be observed in the comparison with 
CAE-SVDD algorithm, applying PCA to reduce features vector dimensionality, leads to better performance 
than the CAE-SVDD structure. Hence the proposed defect detection system could operate well in high 
dimensional spaces. However, the proposed approach has some limitations. As shown in Figure 5, the system 
failed to recognize a defect when the contrast of defected part is similar to the texture of the image. 


Table 1. Experimental results of different approaches on the carpet dataset 


Approach AUROC _ Accuracy _ FI score 
Liu et al. [18] 0.78 - - 
Wang et al. [19] 0.94 0.71 0.88 
Bergmann et al. [32] 0.87 0.67 0.87 
Bergmann et al. [32] 0.59 0.50 0.54 
CAE-SVDD 0.71 0.60 0.67 
CAE-PCA-SVDD 0.98 0.94 0.97 


Figure 5. A false negative case 


4. CONCLUSION 

The current paper proposes automatic defect detection system that attempts to mitigate the lack of 
representative samples of defected images. Hence it can be applied in cases where significant amount of 
normal data is available, while the defected class is characterized by few samples. The suggested approach 
comprises three components: convolutional autoencoder for image features extraction, then PCA is applied to 
reduce dimensionality of features vector, and finally SVDD is implemented to classify images. The 
motivation behind this structure is to build automatic defect detection system that is able to handle effectively 
imbalanced data. The proposed method has proved to be efficient in experiment developed on carpet dataset 
from MVTec AD. It is important to note that SVDD parameters (C and the width a ) vary from one dataset to 
another. In this work C was defined as C=1, while o was chosen through cross-validation to find best 
accuracy. As a future work, we plan to evaluate the proposed approach on different datasets, and also utilize 
an automatic method to determine the parameters: (C,o), which will facilitate the application of our 
approach. Furthermore we will investigate further techniques to improve the ability of the proposed system to 
handle texture features. 
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